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ABSTRACT 


!r  -  vt 

The  following  is  a  progress  report  for  the  period  30  June  1982  to 
29  June  1983  on  Grant  Number  AF0SR-81-0188 ,  "Simply  Instrumentable  and 
Optimal  Digitization  of  Analog  Information  Sources"^  William  A.  Pearlman, 
principal  investigator.  s  New  techniques  for  encoding  sources  with  and 
without  memory  are  described  in  Section  1  and  the  papers,  presentations, 
and  theses  supported  by  the  research  are  listed  in  Section  2.  Three  new 
optimal  coding  methods,  generalizing  previous  ones,  have  been  discovered 
for  stationary  Gaussian  sources  and  the  squared-error  distortion  measure. 
These  methods  ,  which  store  codewords  on  trees  and  trellises  and  utilize 
systematic  search  algorithms,  are  being  applied  to  actual  speech  and  image 
data.  Also,  the  code  storage  requirements  are  not  large,  as  this  research 
has  shown  that  small  code  letter  alphabets  give  nearly  optimal  results. 
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0.  INTRODUCTION 

The  following  is  a  report  on  the  activity  and  progress  for  the  period 
30  June  1982  to  29  June  1983  on  the  grant  entitled,  "Simply  Instrumentable 
and  Optimal  Digitization  of  Analog  Information  Sources",  Grant  Number  AFOSR- 
81-0188,  by  William  A.  Pearlman,  principal  investigator.  For  the  sake  of 
completeness  and  establishing  the  framework,  we  describe  some  work  which  was 
begun  in  the  previous  year  and  carried  over  to  the  current  period.  In 
Section  1,  we  present  past  and  current  work  on  techniques  for  encoding  sources 
with  and  without  memory;  and  in  Section  2  we  list  the  papers,  presentations 
and  theses  supported  by  the  research. 


1.  SUMMARY  OF  PROGRESS 

A.  Memoryless  Sources 
A.l  R^^D)  Calculations 

We  undertook  the  calculation  of  the  R^CD),  the  constrained-size  repro¬ 
duction  alphabet  rate-distortion  function,  for  the  Laplacian  source  with 
squared-error  distortion  measure  to  complement  the  previous  Gaussian  results. 
The  calculation  of  this  function  gives  necessary  parameters  for  accomplishing 
coding  simulations  and  furnishes  absolute  bounds  on  coding  performance  with 
a  size  M  reproduction  alphabet.  As  with  the  Gaussian  case,  we  confirm  the 
theory  in  that  performance  close  to  R(D),  the  rate-distortion  bound,  can  be 
obtained  with  reasonably  small  values  of  L,  the  size  of  amplitude  range  par¬ 
tition,  and  M.  The  value  of  M  for  Laplacian  needs  to  be  larger  than  that 
for  Gaussian  in  order  to  obtain  the  same  closeness  to  R(D)  at  the  same  rate. 
These  calculations  were  used  in  two  other  tasks  in  memoryless  source  coding 
(to  be  described)  and  will  be  important  for  tasks  yet  to  be  undertaken  in 
coding  sources  with  memory. 


A. 2  Sliding-Block  and  Random  Source  Coding 

With  the  source  coding  parameters  furnished  by  the  Rj^D)  calculations 
we  were  able  to  simulate  at  the  rate  of  one  bit  per  source  symbol  new  sliding 
block  codes  and  random  trellis  codes  for  Gaussian  and  Laplacian  sources  with 
the  squared-error  distortion  measure.  These  codes  used  only  a  four  letter 
reproduction  alphabet  and  achieved  performance  either  comparable  or  superior 
to  previous  simulations  with  continuous  reproduction  alphabets.  The  method 
and  results  are  published  in  the  IEEE  Transactions  on  Communications  (Paper 
A.l  in  Section  II). 
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A. 3  Value  and  Probability  Constraints 

The  theory,  where  only  the  size  of  the  reproduction  alphabet 

is  fixed,  leads  naturally  toward  imposing  further  constraints  on  the  repro¬ 
duction  alphabet.  These  constraints  are  specifically  fixing  the  values  and 
probabilities  of  the  letters  in  this  alphabet.  The  rate-distortion  functions 
defined  with  these  reproduction  alphabet  constraints  are  absolute  bounds  in 
performance  when  encoding  under  these  same  constraints  on  choice  of  the 
reproduction  alphabet.  Furthermore,  fixing  either  the  values  or  probabilities 
or  both  obviates  a  calculation  of  a  rate-distortion  function  to  find  the 
missing  coding  parameters.  Reasonable  choices  of  reproduction  values  and 
probabilities  which  come  to  mind  are  those  of  the  optimum  M-level  quantizer. 
These  parameters  are  easily  found  through  existing  published  tables  or  cal¬ 
culated  by  straightforward  numerical  algorithms.  The  ultimate  question  is 
whether  these  additional  constraints  on  the  reproduction  alphabet  impose 
intolerable  losses  in  coding  performance.  Through  calculations  of  the 
appropriately  constrained  alphabet  rate-distortion  functions  for  Gaussian 
and  Laplacian  sources  with  squared-error  distortion,  we  have  found  that  the 
loss  in  performance  is  quite  small  under  these  additional  constraints  on  the 
reproduction  alphabet.  Various  aspects  of  the  theory  and  results  have  been 
presented  at  the  1982  IEEE  International  Conference  on  Communications,  whose 
Proceedings  contains  the  full  paper  (see  B.2  in  Section  2),  and  at  the  1982 
IEEE  International  Symposium  on  Information  Theory  (see  C.3  in  Section  2). 

A  more  detailed  exposition  has  been  submitted  for  publication  review  to  the 
IEEE  Transactions  on  Information  Theory  (see  A. 2,  Section  2). 

A. 4  Source  Mismatch,  Adaptive  Coding,  and  Channel  Errors 

The  source  coding  techniques  in  this  research  are  designed  to  be 
instrumentable.  In  this  spirit*we  have  investigated  aspects  of  constrained 
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alphabet  codes  for  meraoryless  sources  which  often  arise  in  practice  or  imple¬ 
mentation.  The  first  is  the  effect  of  source  mismatch,  which  arises  when 
one  assumes  a  source  probability  distribution  different  from  the  actual  one. 
Mismatch  in  distributional  shape,  variance,  or  a  combination  of  both  are 
possible.  First,  both  constrained  and  unconstrained  codes  seem  to  be  relatively 
insensitive  to  shape  mismatch,  as  shown  through  calculations  on  a  variety  of 
shapes  in  the  generalized  Gaussian  family.  Moreover,  as  long  as  L  and  M, 
the  input  range  partition  and  alphabet  sizes,  are  set  to  give  theoretical 
performance  close  to  the  rate-distortion  bound,  the  alphabet  constrained 
codes  are  no  better  or  worse  than  the  unconstrained  ones  in  their  sensitivity 
to  source  mismatch.  These  results  appear  in  J.  World's  thesis  (see  D.2, 

Section  2) . 

In  the  case  of  channel  errors,  the  conclusion  is  the  same.  If  L  and  M 
are  chosen  appropriately  as  above,  the  alphabet  constrained  codes  have  sensi¬ 
tivity  to  channel  errors  about  equal  to  that  of  unconstrained  codes.  Both 
analysis  and  simulation  confirm  this  conclusion.  The  results  are  reported  in 
H.  Aschmann's  thesis  (see  D.l,  Section  2). 

We  just  completed  a  study  of  coding  of  memoryless  sources  adaptively 
to  overcome  the  deleterious  effects  of  variance  mismatch.  The  model  paradigm 
is  the  periodically  quasi-stationary  driving  noise  of  the  auto-regressive 
source  used  to  model  speech.  We  have  found  the  constrained  alphabet  code  to 
be  slightly  less  robust  and  a  little  more  difficult  to  adapt,  but  giving 
nearly  the  same  performance  except  for  small  mismatch,  in  which  case  the 
unconstrained  code  performs  far  worse.  The  complete  results  appear  in 
A.  Chaddha’s  thesis  (see  D.3,  Section  2). 

We  plan  to  write  one  or  more  papers  to  report  these  results  on  mismatch, 
adaptive  coding,  and  effects  of  channel  errors. 


Sources  with  memory  have  the  greater  potential  payoff  with  efficient 
data  compression  than  sources  without  memory.  Until  now  there  has  been  only 
one  coding  theorem  proved  for  a  source  with  memory  that  yields  an  implement- 
able  encoding  scheme.  This  coding  theorem  applies  only  to  a  Gaussian  auto¬ 
regressive  source  with  squared-error  distortion  above  a  so-called  critical 
rate.  Under  this  grant  we  have  proved  three  new  theorems  which  yield 
implementable  coding  schemes  for  Gaussian  sources  (auto-regressive  or  not), 
with  squared-error  distortion,  applicable  to  all  nonzero  rates.  In  chronolog¬ 
ical  order  of  discovery,  our  first  theorem  proves  that  an  optimal  tree  code 
can  be  constructed  to  operate  on  the  Karhunen-Loeve  (KL)  transform  of  the 
source.  Our  second  theorem  proves  that  there  exists  an  optimal  tree  code 
operating  on  the  actual  time  values  of  the  source.  The  last  one  proves  the 
existence  of  an  optimal  trellis  code  for  the  KL  transform  of  the  source.  The 
proofs  of  all  three  theorems  provide  a  methodology  for  building  a  code  with 
performance  approaching  the  rate-distortion  bound.  The  two  tree  codes  are 
theoretically  not  decodable  because  of  the  requirement  at  the  decoder  of 
storing  a  tree,  the  number  of  branches  of  which  grows  exponentially  with  the 
length  of  the  source  sequence.  In  practice,  however,  one  can  use  knowledge 
of  the  seed  of  a  pseudo-random  number  generator  to  generate  the  reproduction 
values  on  the  tree  branches  at  the  decoder  without  storing  the  tree.  The 
trellis  code,  however,  can  be  decoded  theoretically  and  practically,  because 
the  storage  requirement  increases  only  linearly  and  not  exponentially  with 
the  source  sequence  length.  Encoding  simulations  have  been  undertaken  using 
all  three  methods  given  by  these  theorems.  They  all  give  performance  close 
to  the  rate-distortion  bound.  As  yet  we  see  no  advantage  of  one  method  over 
the  other  in  terms  of  performance  versus  search  intensity  for  the  various 
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Gaussian  sources  we  have  used.  The  results  of  this  work  on  ideally  stationary 
Gaussian  sources  with  memory  are  now  published  in  two  conference  papers 
(Section  2,  B.l  and  B.3).  The  transform  trellis  code  construction  and  the 
coding  theorem  will  be  published  in  the  IEEE  Transactions  on  Information 
Theory  (Section  2,  B.3),  while  the  simulation  results  and  method  of  code  con¬ 
struction  will  be  presented  and  appear  in  full  in  the  conference  record  of 
ICC  '83  (Section  2,  D.4). 

A  great  deal  of  effort  has  been  undertaken  to  apply  these  new  coding 
methods  to  actual  image  and  speech  sources.  In  his  Ph.D.  dissertation 
(Sec.  2,  D.4),  P.  Jakatdar  has  applied  the  transform  tree  technique  toward 
encoding  images.  He  has  compared  the  performance  with  that  of  transform 
quantization  and  has  investigated  theoretically  and  experimentally,  the  effects 
of  channel  errors  on  both  schemes.  The  technique  shows  promise  in  increased 
SNR  over  quantization,  but  needs  to  be  adapted  more  effectively  to  the  chang¬ 
ing  statistics  over  the  image  area.  A  paper  describing  the  method  and  results 
has  been  submitted  for  presentation  and  inclusion  in  the  record  of  ICC  '84 
(sec.  2,  B. 5) . 

The  transform  trellis  code  technique  has  been  used  to  encode  actual 
speech.  Here  some  crude  adaptation  of  the  code  has  been  attempted  to  account 
for  the  quasi-stationary  nature  of  speech.  The  SNR  figures  are  markedly 
superior  to  comparable  ones  in  the  literature.  A  paper  describing  the  method 
and  results  has  been  submitted  for  presentation  at  the  forthcoming  IEEE 
International  Symposium  on  Information  Theory  (Sec.  2,  C.6).  We  can  report 
no  subjective  results  with  this  speech  coding  technique,  as  we  have  no  facil¬ 
ities  for  listening  to  the  coded  speech. 

Much  material  now  written  only  in  P.  Jakatdar's  and  B.  Mazor's  Ph.D. 
dissertations  (Sec.  2,  D.4  and  D.5)  is  suitable  for  journal  publication.  We 
plan  to  write  these  articles  as  soon  as  possible. 
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2 .  PAPERS  i 

j 

Listed  below  are  the  articles  and  conference  presentation  describing 
the  results  of  the  research. 

A.  Journal  Articles 

1.  W.  A.  Pearlman,  "Sliding-Block  and  Random  Source  Coding  with  Constrained 
Size  Reproduction  Alphabets",  IEEE  Transactions  on  Communications,  Vol. 

COM- 30,  pp.  1859-1867,  August  1982. 

j 

2.  W.  A.  Pearlman  and  A.  Chekima,  "Source  Coding  Bounds  Using  Quantizer 

l 

Reproduction  Levels",  submitted  to  the  IEEE  Transactions  on  Information 

Theory.  ! 

3.  B.  Mazor  and  W.  A.  Pearlman,  "A  Trellis  Code  Construction  and  Coding 
Theorem  for  Stationary  Gaussian  Sources",  to  be  published  in  the  IEEE 
Transactions  on  Information  Theory,  November  1983. 

B.  Conference  Proceedings  Articles 

1.  P.  Jakatdar  and  W.  A.  Pearlman,  "Very  Low  Rate  Tree  Coding  of  Stationary 
Gaussian  Sources",  Proceedings  of  the  Sixteenth  Annual  Conference  on 
Information  Sciences  and  Systems,  Princeton  University,  March  1982, 

pp.  559-563. 

2.  W.  A.  Pearlman  and  A.  Chekima,  "Source  Coding  Bounds  with  the  Recon¬ 
struction  Values  of  the  Optimal  Quantizer",  Proceedings  of  the  IEEE 
International  Conference  on  Communications,  Philadelphia,  PA,  June  1982, 
pp.  4H.1.1-4H.1.5. 

3.  B.  Mazor  and  W.  A.  Pearlman,  "An  Optimal  Approach  for  Time-Domain  Tree 
Coding  of  Stationary  Gaussian  Sources",  Proceedings  of  Twentieth  Annual 


Allerton  Conference  on  Communication,  Control  and  Computing,  Monticello 
IL,  October  1982,  pp.  689-696. 


4.  B.  Mazor  and  W.  A.  Pearlman,  "A  Transform  Trellis  Code  for  Stationary 

Gaussian  Sources",  Conference  Record,  1983  IEEE  International  Conference 
on  Communications,  Boston,  MA,  June  1983,  pp.  1090-1094. 
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5.  W.  A.  Pearlman  and  P.  Jakatdar,  "Transform  Tree  Coding  of  Images", 

submitted  to  the  1984  IEEE  International  Conference  on  Communications, 
Amsterdam,  The  Netherlands,  May  14-17,  1984. 

C.  Conference  Presentations 

1.  See  B.l  above 

2.  See  B.2  above 

3.  W.  A.  Pearlman  and  A.  Chekima,  "Encoding  Bounds  for  Analog  Sources 
Using  Optimum  Quantizer  Range  and  Reproduction  Values",  1982  IEEE 
International  Symposium  on  Information  Theory,  Les  Arcs,  France, 

June  1982  (Abstract  in  Conference  Record). 

4.  See  B.3  above 

5.  See  B.4  above 

6.  B.  Mazor  and  W.  A.  Pearlman,  "Transform  Trellis  Coding  of  Speech",  to  be 
presented  at  the  1983  IEEE  International  Symposium  on  Information  Theory, 
St.  Jovite,  Quebec,  Canada,  September  1983. 

7.  See  B.5  above. 

D.  Theses  and  Reports 

1.  H.  R.  Aschmann,  On  Encoding  Algorithms  and  Channel  Errors  in  Trellis 
Codes,  Master  of  Engineering  Project  Report,  ECSE  Dept.,  RPI,  May  1982. 

2.  J.  M.  World,  Mismatch  of  Finite  Alphabet  Source  Encoders,  Master  of 


Engineering  Project  Report,  ECSE  Dept.,  RPI,  August  1982. 
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3.  A.  Chaddha,  Adaptive  Encoding  of  Memoryless  Sources,  Master  of  Science 
Thesis,  ECSE  Dept.,  RPI,  March  1983  (May  1983  Graduation). 

4.  P.  Jakatdar,  Optimal  Encoding  of  Stationary  Gaussian  Sources  with 
Applications  in  Image  Coding,  Ph.D.  Dissertation,  ECSE  Dept.,  RPI, 

January  1983  (May  1983  Graduation) . 

5.  B.  Ma2or,  Optimal  Tree  and  Trellis  Codes  for  Stationary  Gaussian  Sources, 
Ph.D.  Dissertation,  ECSE  Dept.,  RPI,  May  1983. 

6.  A.  Chekima,  Optimal  Coding  of  Continuous,  Memoryless  Sources  with 
Constrained  Reproduction  Alphabets,  Ph.D.  Dissertation,  ECSE  Dept.,  RPI, 
December  1983. 
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